HPCC
nikolays
7/28/20
Everything is executed on single node, average of 10 runs.
Metrics
Some of reported HPCC metrics:
- Average Double-Precision General Matrix Multiplication (DGEMM) Floating-Point Performance
- Embarrassingly parallel, Averaged over all MPI process, MFLOP per Second per MPI process
- (There is also serial metric; measured but not analysed)
- High Performance LINPACK Floating-Point Performance
- Parallel, overall MFLOP per Second
- Fast Fourier Transform (FFTW) Floating-Point Performance
- Parallel, MFLOP per Second
- (There are also serial and embarrassingly parallel metrics; measured but not analysed)
- Parallel Matrix Transpose (PTRANS)
- Parallel, overall MByte per Second
- MPI Random Access
- Parallel, overall MUpdate per Second
- (There are also serial and embarrassingly parallel metrics; measured but not analysed)
- Average STREAM ‘Triad’/ ‘Add’ / ‘Copy’ / ‘Scale’ Memory Bandwidth
- Embarrassingly parallel, Averaged over all MPI process, MByte per Second
- (There is also serial metric; measured but not analysed)
Results
Overall Performance (That is per All MPI Processes)
High Performance LINPACK Floating-Point Performance, MFLOP per Second
| resource | exe_type | cpus | average | stdev |
|---|---|---|---|---|
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/sysblas/intfft | 64 | 46535.57 | 237.15 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/sysblas/intfft | 128 | 47026.24 | 141.35 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/sysblas/intfft | 256 | 45474.51 | 69.34 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/openblas/intfft | 64 | 581802.00 | 3172.57 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/openblas/intfft | 128 | 435367.30 | 991.71 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/openblas/intfft | 256 | 330640.90 | 2516.47 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/armpl/intfft | 64 | 485480.90 | 1243.16 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/armpl/intfft | 128 | 458037.20 | 2048.63 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/armpl/intfft | 256 | 421653.10 | 3126.48 |
| Stampede2 SKX (CPU Intel Xeon Platinum 8160 24(48) cores x2) | impi/mkl | 48 | 988258.60 | 20198.84 |
| Stampede2 KNL (CPU Intel Xeon Phi 7250 68(272) cores x2) | impi/mkl | 68 | 987046.80 | 14078.96 |
| Comet (CPU Intel Xeon E5-2680v3 12 cores x2, Haswell) | mvapich2/mkl | 24 | 515101.60 | 10552.39 |
| Bridges (Intel E5-2695 v3 14 cores x2, Haswell) | impi/mkl | 28 | 554204.50 | 62818.10 |
Fast Fourier Transform (FFTW) Floating-Point Performance, MFLOP per Second
| resource | exe_type | cpus | average | stdev |
|---|---|---|---|---|
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/sysblas/intfft | 64 | 15388.25 | 189.37 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/sysblas/intfft | 128 | 16445.68 | 513.88 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/sysblas/intfft | 256 | 18292.79 | 981.23 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/openblas/intfft | 64 | 15490.95 | 340.47 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/openblas/intfft | 128 | 16954.09 | 226.83 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/openblas/intfft | 256 | 17948.42 | 1564.26 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/armpl/intfft | 64 | 15247.34 | 165.05 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/armpl/intfft | 128 | 15983.20 | 422.39 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/armpl/intfft | 256 | 17292.98 | 4492.64 |
| Stampede2 SKX (CPU Intel Xeon Platinum 8160 24(48) cores x2) | impi/mkl | 48 | 43670.26 | 780.33 |
| Stampede2 KNL (CPU Intel Xeon Phi 7250 68(272) cores x2) | impi/mkl | 68 | 46666.67 | 502.63 |
| Comet (CPU Intel Xeon E5-2680v3 12 cores x2, Haswell) | mvapich2/mkl | 24 | 21765.33 | 1191.43 |
| Bridges (Intel E5-2695 v3 14 cores x2, Haswell) | impi/mkl | 28 | 14499.70 | 304.43 |
Parallel Matrix Transpose (PTRANS), MByte per Second
| resource | exe_type | cpus | average | stdev |
|---|---|---|---|---|
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/sysblas/intfft | 64 | 10720.32 | 326.78 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/sysblas/intfft | 128 | 9841.70 | 136.43 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/sysblas/intfft | 256 | 16847.96 | 2121.25 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/openblas/intfft | 64 | 10654.13 | 279.48 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/openblas/intfft | 128 | 10037.08 | 191.98 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/openblas/intfft | 256 | 18119.24 | 1325.63 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/armpl/intfft | 64 | 10576.29 | 438.45 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/armpl/intfft | 128 | 9855.29 | 260.41 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/armpl/intfft | 256 | 17883.64 | 1688.42 |
| Stampede2 SKX (CPU Intel Xeon Platinum 8160 24(48) cores x2) | impi/mkl | 48 | 18074.71 | 446.00 |
| Stampede2 KNL (CPU Intel Xeon Phi 7250 68(272) cores x2) | impi/mkl | 68 | 13965.83 | 209.40 |
| Comet (CPU Intel Xeon E5-2680v3 12 cores x2, Haswell) | mvapich2/mkl | 24 | 9624.68 | 146.03 |
| Bridges (Intel E5-2695 v3 14 cores x2, Haswell) | impi/mkl | 28 | 7165.57 | 173.95 |
MPI Random Access, MUpdate per Second
| resource | exe_type | cpus | average | stdev |
|---|---|---|---|---|
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/sysblas/intfft | 64 | 259.84 | 3.36 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/sysblas/intfft | 128 | 252.05 | 4.77 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/sysblas/intfft | 256 | 173.59 | 6.85 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/openblas/intfft | 64 | 264.26 | 2.07 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/openblas/intfft | 128 | 248.71 | 5.50 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/openblas/intfft | 256 | 177.20 | 14.11 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/armpl/intfft | 64 | 261.14 | 2.88 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/armpl/intfft | 128 | 253.37 | 3.62 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/armpl/intfft | 256 | 176.40 | 14.10 |
| Stampede2 SKX (CPU Intel Xeon Platinum 8160 24(48) cores x2) | impi/mkl | 48 | 39.27 | 0.50 |
| Stampede2 KNL (CPU Intel Xeon Phi 7250 68(272) cores x2) | impi/mkl | 68 | 3.26 | 0.03 |
| Comet (CPU Intel Xeon E5-2680v3 12 cores x2, Haswell) | mvapich2/mkl | 24 | 40.03 | 0.99 |
| Bridges (Intel E5-2695 v3 14 cores x2, Haswell) | impi/mkl | 28 | 118.39 | 18.94 |
Average Double-Precision General Matrix Multiplication (DGEMM) Floating-Point Performance, MFLOP per Second
| resource | exe_type | cpus | average | stdev |
|---|---|---|---|---|
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/sysblas/intfft | 64 | 57371.47 | 66.41 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/sysblas/intfft | 128 | 61774.21 | 64.72 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/sysblas/intfft | 256 | 64949.94 | 178.61 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/openblas/intfft | 64 | 777500.16 | 2118.86 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/openblas/intfft | 128 | 523424.00 | 4498.68 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/openblas/intfft | 256 | 452975.36 | 3563.49 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/armpl/intfft | 64 | 630826.24 | 3461.07 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/armpl/intfft | 128 | 517666.05 | 1939.81 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/armpl/intfft | 256 | 414587.39 | 3954.54 |
| Stampede2 SKX (CPU Intel Xeon Platinum 8160 24(48) cores x2) | impi/mkl | 48 | 2106614.40 | 77528.00 |
| Stampede2 KNL (CPU Intel Xeon Phi 7250 68(272) cores x2) | impi/mkl | 68 | 334587.27 | 14445.84 |
| Comet (CPU Intel Xeon E5-2680v3 12 cores x2, Haswell) | mvapich2/mkl | 24 | 696314.16 | 13099.94 |
| Bridges (Intel E5-2695 v3 14 cores x2, Haswell) | impi/mkl | 28 | 724319.12 | 13386.87 |
Average STREAM ‘Triad’ Memory Bandwidth, MByte per Second
| resource | exe_type | cpus | average | stdev |
|---|---|---|---|---|
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/sysblas/intfft | 64 | 157604.12 | 1781.38 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/sysblas/intfft | 128 | 170510.52 | 9350.40 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/sysblas/intfft | 256 | 173858.07 | 56420.11 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/openblas/intfft | 64 | 158843.60 | 2085.14 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/openblas/intfft | 128 | 167578.30 | 9121.34 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/openblas/intfft | 256 | 154225.42 | 20570.22 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/armpl/intfft | 64 | 157605.30 | 2097.07 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/armpl/intfft | 128 | 169241.35 | 6080.71 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/armpl/intfft | 256 | 147068.76 | 1834.26 |
| Stampede2 SKX (CPU Intel Xeon Platinum 8160 24(48) cores x2) | impi/mkl | 48 | 149741.62 | 2748.06 |
| Stampede2 KNL (CPU Intel Xeon Phi 7250 68(272) cores x2) | impi/mkl | 68 | 315214.94 | 23818.48 |
| Comet (CPU Intel Xeon E5-2680v3 12 cores x2, Haswell) | mvapich2/mkl | 24 | 89959.69 | 872.67 |
| Bridges (Intel E5-2695 v3 14 cores x2, Haswell) | impi/mkl | 28 | 87356.19 | 532.06 |
Average STREAM ‘Add’ Memory Bandwidth, MByte per Second
| resource | exe_type | cpus | average | stdev |
|---|---|---|---|---|
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/sysblas/intfft | 64 | 151023.98 | 1806.49 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/sysblas/intfft | 128 | 169686.86 | 3973.33 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/sysblas/intfft | 256 | 179711.27 | 68912.78 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/openblas/intfft | 64 | 150678.80 | 1669.13 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/openblas/intfft | 128 | 173855.87 | 6920.44 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/openblas/intfft | 256 | 169258.83 | 32883.04 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/armpl/intfft | 64 | 152415.90 | 7146.12 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/armpl/intfft | 128 | 170028.17 | 8595.58 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/armpl/intfft | 256 | 155504.82 | 2356.32 |
| Stampede2 SKX (CPU Intel Xeon Platinum 8160 24(48) cores x2) | impi/mkl | 48 | 149137.05 | 2318.96 |
| Stampede2 KNL (CPU Intel Xeon Phi 7250 68(272) cores x2) | impi/mkl | 68 | 316756.66 | 21596.90 |
| Comet (CPU Intel Xeon E5-2680v3 12 cores x2, Haswell) | mvapich2/mkl | 24 | 89668.49 | 981.04 |
| Bridges (Intel E5-2695 v3 14 cores x2, Haswell) | impi/mkl | 28 | 86956.61 | 489.94 |
Average STREAM ‘Copy’ Memory Bandwidth, MByte per Second
| resource | exe_type | cpus | average | stdev |
|---|---|---|---|---|
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/sysblas/intfft | 64 | 129057.8 | 5813.76 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/sysblas/intfft | 128 | 143716.5 | 5157.72 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/sysblas/intfft | 256 | 163230.1 | 65730.92 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/openblas/intfft | 64 | 132548.7 | 6335.55 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/openblas/intfft | 128 | 137553.0 | 1679.58 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/openblas/intfft | 256 | 168196.8 | 37854.83 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/armpl/intfft | 64 | 133553.3 | 6625.87 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/armpl/intfft | 128 | 139392.5 | 1139.72 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/armpl/intfft | 256 | 146985.1 | 12762.55 |
| Stampede2 SKX (CPU Intel Xeon Platinum 8160 24(48) cores x2) | impi/mkl | 48 | 132971.1 | 2353.97 |
| Stampede2 KNL (CPU Intel Xeon Phi 7250 68(272) cores x2) | impi/mkl | 68 | 256390.0 | 7269.67 |
| Comet (CPU Intel Xeon E5-2680v3 12 cores x2, Haswell) | mvapich2/mkl | 24 | 95779.5 | 306.79 |
| Bridges (Intel E5-2695 v3 14 cores x2, Haswell) | impi/mkl | 28 | 100068.5 | 385.94 |
Average STREAM ‘Scale’ Memory Bandwidth, MByte per Second
| resource | exe_type | cpus | average | stdev |
|---|---|---|---|---|
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/sysblas/intfft | 64 | 145269.72 | 3584.16 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/sysblas/intfft | 128 | 151182.25 | 4360.15 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/sysblas/intfft | 256 | 152769.40 | 50916.58 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/openblas/intfft | 64 | 143432.81 | 923.46 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/openblas/intfft | 128 | 149246.44 | 4300.89 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/openblas/intfft | 256 | 144922.80 | 24866.73 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/armpl/intfft | 64 | 146640.08 | 2347.68 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/armpl/intfft | 128 | 145871.47 | 6602.79 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/armpl/intfft | 256 | 138834.45 | 2532.77 |
| Stampede2 SKX (CPU Intel Xeon Platinum 8160 24(48) cores x2) | impi/mkl | 48 | 131272.56 | 2988.52 |
| Stampede2 KNL (CPU Intel Xeon Phi 7250 68(272) cores x2) | impi/mkl | 68 | 270395.12 | 20602.43 |
| Comet (CPU Intel Xeon E5-2680v3 12 cores x2, Haswell) | mvapich2/mkl | 24 | 79454.04 | 361.65 |
| Bridges (Intel E5-2695 v3 14 cores x2, Haswell) | impi/mkl | 28 | 77124.61 | 480.41 |
Performance per Core (that is MPI Process)
High Performance LINPACK Floating-Point Performance, MFLOP per Second
| resource | exe_type | cpus | average | stdev |
|---|---|---|---|---|
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/sysblas/intfft | 64 | 727.12 | 237.15 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/sysblas/intfft | 128 | 367.39 | 141.35 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/sysblas/intfft | 256 | 177.63 | 69.34 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/openblas/intfft | 64 | 9090.66 | 3172.57 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/openblas/intfft | 128 | 3401.31 | 991.71 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/openblas/intfft | 256 | 1291.57 | 2516.47 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/armpl/intfft | 64 | 7585.64 | 1243.16 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/armpl/intfft | 128 | 3578.42 | 2048.63 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/armpl/intfft | 256 | 1647.08 | 3126.48 |
| Stampede2 SKX (CPU Intel Xeon Platinum 8160 24(48) cores x2) | impi/mkl | 48 | 20588.72 | 20198.84 |
| Stampede2 KNL (CPU Intel Xeon Phi 7250 68(272) cores x2) | impi/mkl | 68 | 14515.39 | 14078.96 |
| Comet (CPU Intel Xeon E5-2680v3 12 cores x2, Haswell) | mvapich2/mkl | 24 | 21462.57 | 10552.39 |
| Bridges (Intel E5-2695 v3 14 cores x2, Haswell) | impi/mkl | 28 | 19793.02 | 62818.10 |
Fast Fourier Transform (FFTW) Floating-Point Performance, MFLOP per Second
| resource | exe_type | cpus | average | stdev |
|---|---|---|---|---|
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/sysblas/intfft | 64 | 240.44 | 189.37 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/sysblas/intfft | 128 | 128.48 | 513.88 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/sysblas/intfft | 256 | 71.46 | 981.23 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/openblas/intfft | 64 | 242.05 | 340.47 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/openblas/intfft | 128 | 132.45 | 226.83 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/openblas/intfft | 256 | 70.11 | 1564.26 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/armpl/intfft | 64 | 238.24 | 165.05 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/armpl/intfft | 128 | 124.87 | 422.39 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/armpl/intfft | 256 | 67.55 | 4492.64 |
| Stampede2 SKX (CPU Intel Xeon Platinum 8160 24(48) cores x2) | impi/mkl | 48 | 909.80 | 780.33 |
| Stampede2 KNL (CPU Intel Xeon Phi 7250 68(272) cores x2) | impi/mkl | 68 | 686.27 | 502.63 |
| Comet (CPU Intel Xeon E5-2680v3 12 cores x2, Haswell) | mvapich2/mkl | 24 | 906.89 | 1191.43 |
| Bridges (Intel E5-2695 v3 14 cores x2, Haswell) | impi/mkl | 28 | 517.85 | 304.43 |
Parallel Matrix Transpose (PTRANS), MByte per Second
| resource | exe_type | cpus | average | stdev |
|---|---|---|---|---|
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/sysblas/intfft | 64 | 167.50 | 326.78 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/sysblas/intfft | 128 | 76.89 | 136.43 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/sysblas/intfft | 256 | 65.81 | 2121.25 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/openblas/intfft | 64 | 166.47 | 279.48 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/openblas/intfft | 128 | 78.41 | 191.98 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/openblas/intfft | 256 | 70.78 | 1325.63 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/armpl/intfft | 64 | 165.25 | 438.45 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/armpl/intfft | 128 | 76.99 | 260.41 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/armpl/intfft | 256 | 69.86 | 1688.42 |
| Stampede2 SKX (CPU Intel Xeon Platinum 8160 24(48) cores x2) | impi/mkl | 48 | 376.56 | 446.00 |
| Stampede2 KNL (CPU Intel Xeon Phi 7250 68(272) cores x2) | impi/mkl | 68 | 205.38 | 209.40 |
| Comet (CPU Intel Xeon E5-2680v3 12 cores x2, Haswell) | mvapich2/mkl | 24 | 401.03 | 146.03 |
| Bridges (Intel E5-2695 v3 14 cores x2, Haswell) | impi/mkl | 28 | 255.91 | 173.95 |
MPI Random Access, MUpdate per Second
| resource | exe_type | cpus | average | stdev |
|---|---|---|---|---|
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/sysblas/intfft | 64 | 4.06 | 3.36 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/sysblas/intfft | 128 | 1.97 | 4.77 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/sysblas/intfft | 256 | 0.68 | 6.85 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/openblas/intfft | 64 | 4.13 | 2.07 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/openblas/intfft | 128 | 1.94 | 5.50 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/openblas/intfft | 256 | 0.69 | 14.11 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/armpl/intfft | 64 | 4.08 | 2.88 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/armpl/intfft | 128 | 1.98 | 3.62 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/armpl/intfft | 256 | 0.69 | 14.10 |
| Stampede2 SKX (CPU Intel Xeon Platinum 8160 24(48) cores x2) | impi/mkl | 48 | 0.82 | 0.50 |
| Stampede2 KNL (CPU Intel Xeon Phi 7250 68(272) cores x2) | impi/mkl | 68 | 0.05 | 0.03 |
| Comet (CPU Intel Xeon E5-2680v3 12 cores x2, Haswell) | mvapich2/mkl | 24 | 1.67 | 0.99 |
| Bridges (Intel E5-2695 v3 14 cores x2, Haswell) | impi/mkl | 28 | 4.23 | 18.94 |
Average Double-Precision General Matrix Multiplication (DGEMM) Floating-Point Performance, MFLOP per Second
| resource | exe_type | cpus | average | stdev |
|---|---|---|---|---|
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/sysblas/intfft | 64 | 896.43 | 66.41 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/sysblas/intfft | 128 | 482.61 | 64.72 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/sysblas/intfft | 256 | 253.71 | 178.61 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/openblas/intfft | 64 | 12148.44 | 2118.86 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/openblas/intfft | 128 | 4089.25 | 4498.68 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/openblas/intfft | 256 | 1769.43 | 3563.49 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/armpl/intfft | 64 | 9856.66 | 3461.07 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/armpl/intfft | 128 | 4044.27 | 1939.81 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/armpl/intfft | 256 | 1619.48 | 3954.54 |
| Stampede2 SKX (CPU Intel Xeon Platinum 8160 24(48) cores x2) | impi/mkl | 48 | 43887.80 | 77528.00 |
| Stampede2 KNL (CPU Intel Xeon Phi 7250 68(272) cores x2) | impi/mkl | 68 | 4920.40 | 14445.84 |
| Comet (CPU Intel Xeon E5-2680v3 12 cores x2, Haswell) | mvapich2/mkl | 24 | 29013.09 | 13099.94 |
| Bridges (Intel E5-2695 v3 14 cores x2, Haswell) | impi/mkl | 28 | 25868.54 | 13386.87 |
Average STREAM ‘Triad’ Memory Bandwidth, MByte per Second
| resource | exe_type | cpus | average | stdev |
|---|---|---|---|---|
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/sysblas/intfft | 64 | 2462.56 | 1781.38 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/sysblas/intfft | 128 | 1332.11 | 9350.40 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/sysblas/intfft | 256 | 679.13 | 56420.11 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/openblas/intfft | 64 | 2481.93 | 2085.14 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/openblas/intfft | 128 | 1309.21 | 9121.34 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/openblas/intfft | 256 | 602.44 | 20570.22 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/armpl/intfft | 64 | 2462.58 | 2097.07 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/armpl/intfft | 128 | 1322.20 | 6080.71 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/armpl/intfft | 256 | 574.49 | 1834.26 |
| Stampede2 SKX (CPU Intel Xeon Platinum 8160 24(48) cores x2) | impi/mkl | 48 | 3119.62 | 2748.06 |
| Stampede2 KNL (CPU Intel Xeon Phi 7250 68(272) cores x2) | impi/mkl | 68 | 4635.51 | 23818.48 |
| Comet (CPU Intel Xeon E5-2680v3 12 cores x2, Haswell) | mvapich2/mkl | 24 | 3748.32 | 872.67 |
| Bridges (Intel E5-2695 v3 14 cores x2, Haswell) | impi/mkl | 28 | 3119.86 | 532.06 |
Average STREAM ‘Add’ Memory Bandwidth, MByte per Second
| resource | exe_type | cpus | average | stdev |
|---|---|---|---|---|
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/sysblas/intfft | 64 | 2359.75 | 1806.49 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/sysblas/intfft | 128 | 1325.68 | 3973.33 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/sysblas/intfft | 256 | 702.00 | 68912.78 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/openblas/intfft | 64 | 2354.36 | 1669.13 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/openblas/intfft | 128 | 1358.25 | 6920.44 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/openblas/intfft | 256 | 661.17 | 32883.04 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/armpl/intfft | 64 | 2381.50 | 7146.12 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/armpl/intfft | 128 | 1328.35 | 8595.58 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/armpl/intfft | 256 | 607.44 | 2356.32 |
| Stampede2 SKX (CPU Intel Xeon Platinum 8160 24(48) cores x2) | impi/mkl | 48 | 3107.02 | 2318.96 |
| Stampede2 KNL (CPU Intel Xeon Phi 7250 68(272) cores x2) | impi/mkl | 68 | 4658.19 | 21596.90 |
| Comet (CPU Intel Xeon E5-2680v3 12 cores x2, Haswell) | mvapich2/mkl | 24 | 3736.19 | 981.04 |
| Bridges (Intel E5-2695 v3 14 cores x2, Haswell) | impi/mkl | 28 | 3105.59 | 489.94 |
Average STREAM ‘Copy’ Memory Bandwidth, MByte per Second
| resource | exe_type | cpus | average | stdev |
|---|---|---|---|---|
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/sysblas/intfft | 64 | 2016.53 | 5813.76 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/sysblas/intfft | 128 | 1122.79 | 5157.72 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/sysblas/intfft | 256 | 637.62 | 65730.92 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/openblas/intfft | 64 | 2071.07 | 6335.55 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/openblas/intfft | 128 | 1074.63 | 1679.58 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/openblas/intfft | 256 | 657.02 | 37854.83 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/armpl/intfft | 64 | 2086.77 | 6625.87 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/armpl/intfft | 128 | 1089.00 | 1139.72 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/armpl/intfft | 256 | 574.16 | 12762.55 |
| Stampede2 SKX (CPU Intel Xeon Platinum 8160 24(48) cores x2) | impi/mkl | 48 | 2770.23 | 2353.97 |
| Stampede2 KNL (CPU Intel Xeon Phi 7250 68(272) cores x2) | impi/mkl | 68 | 3770.44 | 7269.67 |
| Comet (CPU Intel Xeon E5-2680v3 12 cores x2, Haswell) | mvapich2/mkl | 24 | 3990.81 | 306.79 |
| Bridges (Intel E5-2695 v3 14 cores x2, Haswell) | impi/mkl | 28 | 3573.87 | 385.94 |
Average STREAM ‘Scale’ Memory Bandwidth, MByte per Second
| resource | exe_type | cpus | average | stdev |
|---|---|---|---|---|
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/sysblas/intfft | 64 | 2269.84 | 3584.16 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/sysblas/intfft | 128 | 1181.11 | 4360.15 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/sysblas/intfft | 256 | 596.76 | 50916.58 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/openblas/intfft | 64 | 2241.14 | 923.46 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/openblas/intfft | 128 | 1165.99 | 4300.89 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/openblas/intfft | 256 | 566.10 | 24866.73 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/armpl/intfft | 64 | 2291.25 | 2347.68 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/armpl/intfft | 128 | 1139.62 | 6602.79 |
| Koinu (CPU ARM ThunderX2 32(128) cores x2) | openmpi/armpl/intfft | 256 | 542.32 | 2532.77 |
| Stampede2 SKX (CPU Intel Xeon Platinum 8160 24(48) cores x2) | impi/mkl | 48 | 2734.84 | 2988.52 |
| Stampede2 KNL (CPU Intel Xeon Phi 7250 68(272) cores x2) | impi/mkl | 68 | 3976.40 | 20602.43 |
| Comet (CPU Intel Xeon E5-2680v3 12 cores x2, Haswell) | mvapich2/mkl | 24 | 3310.58 | 361.65 |
| Bridges (Intel E5-2695 v3 14 cores x2, Haswell) | impi/mkl | 28 | 2754.45 | 480.41 |
LS0tCnRpdGxlOiAiSFBDQyIKYXV0aG9yOiBuaWtvbGF5cwpkYXRlOiA3LzI4LzIwCm91dHB1dDogaHRtbF9ub3RlYm9vawotLS0KCmBgYHtyIHNldHVwLCBpbmNsdWRlPUZBTFNFfQprbml0cjo6b3B0c19jaHVuayRzZXQoZWNobyA9IFRSVUUpCgpsaWJyYXJ5KGdncGxvdDIpCmxpYnJhcnkoZHBseXIpCmxpYnJhcnkoY293cGxvdCkKbGlicmFyeShrbml0cikKbGlicmFyeShrYWJsZUV4dHJhKQpgYGAKCmBgYHtyIGluY2x1ZGU9RkFMU0V9CiMgUmVhZCBkYXRhCmhwY2MgPC0gcmVhZC5jc3YoImRhdGEvaHBjYy5jc3YiLCBzdHJpbmdzQXNGYWN0b3JzPUYpICU+JQogICAgc2VsZWN0KC1YKSAlPiUgZmlsdGVyKG1ldHJpYyAhPSAiQXBwOlZlcnNpb24iKSAlPiUKICAgIG11dGF0ZSgKICAgICAgICB2YWx1ZT1hcy5udW1lcmljKHZhbHVlKSwKICAgICAgICBleGVfdHlwZT1mYWN0b3IoZXhlX3R5cGUpLAogICAgICAgIHByb2JsZW1fc2l6ZT1mYWN0b3IocHJvYmxlbV9zaXplKSwKICAgICAgICBtZXRyaWNfdHlwZT1mYWN0b3IobWV0cmljX3R5cGUpLAogICAgICAgIG1ldHJpYz1mYWN0b3IobWV0cmljKSwKICAgICAgICB1bml0cz1mYWN0b3IodW5pdHMpLAogICAgICAgIHJlc291cmNlPWZhY3RvcihyZXNvdXJjZSwgYygKICAgICAgICAgICAgIktvaW51IChDUFUgQVJNIFRodW5kZXJYMiAzMigxMjgpIGNvcmVzIHgyKSIsCiAgICAgICAgICAgICJTdGFtcGVkZTIgU0tYIChDUFUgSW50ZWwgWGVvbiBQbGF0aW51bSA4MTYwIDI0KDQ4KSBjb3JlcyB4MikiLAogICAgICAgICAgICAiU3RhbXBlZGUyIEtOTCAoQ1BVIEludGVsIFhlb24gUGhpIDcyNTAgNjgoMjcyKSBjb3JlcyB4MikiLAogICAgICAgICAgICAiQ29tZXQgKENQVSBJbnRlbCBYZW9uIEU1LTI2ODB2MyAxMiBjb3JlcyB4MiwgSGFzd2VsbCkiLAogICAgICAgICAgICAiQnJpZGdlcyAoSW50ZWwgRTUtMjY5NSB2MyAxNCBjb3JlcyB4MiwgSGFzd2VsbCkiKSksCiAgICAgICAgZXhlX3R5cGU9ZmFjdG9yKGV4ZV90eXBlLCBjKAogICAgICAgICAgICAib3Blbm1waS9zeXNibGFzL2ludGZmdCIsCiAgICAgICAgICAgICJvcGVubXBpL29wZW5ibGFzL2ludGZmdCIsCiAgICAgICAgICAgICJvcGVubXBpL2FybXBsL2ludGZmdCIsCiAgICAgICAgICAgICJpbXBpL21rbCIsICJtdmFwaWNoMi9ta2wiKSkKICAgICkKI2RwdXQodW5pcXVlKGhwY2MkZXhlX3R5cGUpKQojaGVhZChocGNjKQpgYGAKCmBgYHtyIGluY2x1ZGU9RkFMU0V9CiMgc3RhdGlzdGljcyB3ZSBhcmUgaW50ZXJlc3RlZCBpbgojZHB1dCh1bmlxdWUoaHBjYyRtZXRyaWNbaHBjYyRtZXRyaWNfdHlwZT09InN0YXRpc3RpYyJdKSkKc3RhdHNfb3ZlcmFsbCA8LSBjKAogICAgIkhpZ2ggUGVyZm9ybWFuY2UgTElOUEFDSyBGbG9hdGluZy1Qb2ludCBQZXJmb3JtYW5jZSIsCiAgICAiRmFzdCBGb3VyaWVyIFRyYW5zZm9ybSAoRkZUVykgRmxvYXRpbmctUG9pbnQgUGVyZm9ybWFuY2UiLAogICAgIlBhcmFsbGVsIE1hdHJpeCBUcmFuc3Bvc2UgKFBUUkFOUykiLAogICAgIk1QSSBSYW5kb20gQWNjZXNzIgopCnN0YXRzX3Blcl9jb3JlIDwtIGMoCiAgICAiQXZlcmFnZSBEb3VibGUtUHJlY2lzaW9uIEdlbmVyYWwgTWF0cml4IE11bHRpcGxpY2F0aW9uIChER0VNTSkgRmxvYXRpbmctUG9pbnQgUGVyZm9ybWFuY2UiLAogICAgIkF2ZXJhZ2UgU1RSRUFNICdUcmlhZCcgTWVtb3J5IEJhbmR3aWR0aCIsCiAgICAiQXZlcmFnZSBTVFJFQU0gJ0FkZCcgTWVtb3J5IEJhbmR3aWR0aCIsCiAgICAiQXZlcmFnZSBTVFJFQU0gJ0NvcHknIE1lbW9yeSBCYW5kd2lkdGgiLAogICAgIkF2ZXJhZ2UgU1RSRUFNICdTY2FsZScgTWVtb3J5IEJhbmR3aWR0aCIKKQp1bml0cyA8LSBocGNjICU+JSBncm91cF9ieShtZXRyaWMpICU+JSBzdW1tYXJpc2UodW5pdHM9YXMuY2hhcmFjdGVyKGZpcnN0KHVuaXRzKSkpICU+JQogICAgdW5ncm91cCgpCnVuaXRzJHVuaXRzW2lzLm5hKHVuaXRzJHVuaXRzKV08LSIiCmBgYAoKRXZlcnl0aGluZyBpcyBleGVjdXRlZCBvbiBzaW5nbGUgbm9kZSwgYXZlcmFnZSBvZiAxMCBydW5zLgoKIyMgTWV0cmljcwoKU29tZSBvZiByZXBvcnRlZCBIUENDIG1ldHJpY3M6CgogKiBBdmVyYWdlIERvdWJsZS1QcmVjaXNpb24gR2VuZXJhbCBNYXRyaXggTXVsdGlwbGljYXRpb24gKERHRU1NKSBGbG9hdGluZy1Qb2ludCBQZXJmb3JtYW5jZQogICArIEVtYmFycmFzc2luZ2x5IHBhcmFsbGVsLCBBdmVyYWdlZCBvdmVyIGFsbCBNUEkgcHJvY2VzcywgTUZMT1AgcGVyIFNlY29uZCBwZXIgTVBJIHByb2Nlc3MKICAgKyAoVGhlcmUgaXMgYWxzbyBzZXJpYWwgbWV0cmljOyBtZWFzdXJlZCBidXQgbm90IGFuYWx5c2VkKQogKiBIaWdoIFBlcmZvcm1hbmNlIExJTlBBQ0sgRmxvYXRpbmctUG9pbnQgUGVyZm9ybWFuY2UKICAgKyBQYXJhbGxlbCwgb3ZlcmFsbCBNRkxPUCBwZXIgU2Vjb25kCiAqIEZhc3QgRm91cmllciBUcmFuc2Zvcm0gKEZGVFcpIEZsb2F0aW5nLVBvaW50IFBlcmZvcm1hbmNlCiAgICsgUGFyYWxsZWwsIE1GTE9QIHBlciBTZWNvbmQKICAgKyAoVGhlcmUgYXJlIGFsc28gc2VyaWFsIGFuZCBlbWJhcnJhc3NpbmdseSBwYXJhbGxlbCBtZXRyaWNzOyBtZWFzdXJlZCBidXQgbm90IGFuYWx5c2VkKQogKiBQYXJhbGxlbCBNYXRyaXggVHJhbnNwb3NlIChQVFJBTlMpCiAgICsgUGFyYWxsZWwsIG92ZXJhbGwgTUJ5dGUgcGVyIFNlY29uZAogKiBNUEkgUmFuZG9tIEFjY2VzcwogICArIFBhcmFsbGVsLCBvdmVyYWxsIE1VcGRhdGUgcGVyIFNlY29uZAogICArIChUaGVyZSBhcmUgYWxzbyBzZXJpYWwgYW5kIGVtYmFycmFzc2luZ2x5IHBhcmFsbGVsIG1ldHJpY3M7IG1lYXN1cmVkIGJ1dCBub3QgYW5hbHlzZWQpCiAqIEF2ZXJhZ2UgU1RSRUFNICdUcmlhZCcvICdBZGQnIC8gJ0NvcHknIC8gJ1NjYWxlJyBNZW1vcnkgQmFuZHdpZHRoCiAgICsgRW1iYXJyYXNzaW5nbHkgcGFyYWxsZWwsIEF2ZXJhZ2VkIG92ZXIgYWxsIE1QSSBwcm9jZXNzLCBNQnl0ZSBwZXIgU2Vjb25kCiAgICsgKFRoZXJlIGlzIGFsc28gc2VyaWFsIG1ldHJpYzsgbWVhc3VyZWQgYnV0IG5vdCBhbmFseXNlZCkKCiMjIFJlc3VsdHMKIyMjIE92ZXJhbGwgUGVyZm9ybWFuY2UgKFRoYXQgaXMgcGVyIEFsbCBNUEkgUHJvY2Vzc2VzKQoKYGBge3IgcmVzdWx0cz0nYXNpcycsIGVjaG89RkFMU0UsIGZpZy53aWR0aCA9IDExLCBmaWcuaGVpZ2h0ID0gNn0KZm9yKHN0YXQgaW4gc3RhdHNfb3ZlcmFsbCkgewogICAgc3RhdF91bml0cyA8LSB1bml0cyR1bml0c1t1bml0cyRtZXRyaWM9PXN0YXRdCiAgICBpZihzdGF0X3VuaXRzPT0iIikgewogICAgICAgIGNhcCA8LSBzdGF0CiAgICB9IGVsc2UgewogICAgICAgIGNhcCA8LSBwYXN0ZTAoc3RhdCwgIiwgIiwgc3RhdF91bml0cykKICAgIH0KICAgIGNhdChwYXN0ZTAoIjxoND4iLGNhcCwiPC9oND5cbiIpKQogICAgcGVyZiA8LSBocGNjICU+JSBmaWx0ZXIobWV0cmljID09IHN0YXQpICU+JQogICAgICAgIGdyb3VwX2J5KHJlc291cmNlLGV4ZV90eXBlLGNwdXMpICU+JQogICAgICAgIHN1bW1hcmlzZShhdmVyYWdlPW1lYW4odmFsdWUpLHN0ZGV2PXNkKHZhbHVlKSkgJT4lCiAgICAgICAgdW5ncm91cCgpICU+JSBtdXRhdGUoY29uZj1wYXN0ZShyZXNvdXJjZSxleGVfdHlwZSxjcHVzLCBzZXAgPSAiLCAiKSkgJT4lCiAgICAgICAgbXV0YXRlKGNvbmY9ZmFjdG9yKGNvbmYsIGxldmVscyA9IHJldihjb25mKSkpCiAgICBwcmludChnZ3Bsb3QocGVyZiwgYWVzKHg9Y29uZiwgeT1hdmVyYWdlKSkgKwogICAgICAgIGdlb21fYmFyKHN0YXQ9ImlkZW50aXR5IiwgZmlsbD0icm95YWxibHVlIiwgY29sb3I9InJveWFsYmx1ZTQiKSArCiAgICAgICAgZ2VvbV9lcnJvcmJhcihhZXMoeW1pbj1hdmVyYWdlLXN0ZGV2LCB5bWF4PWF2ZXJhZ2Urc3RkZXYpLCB3aWR0aD0uMikrCiAgICAgICAgY29vcmRfZmxpcCgpK3lsYWIoY2FwKSkKICAgIHByaW50KGthYmxlKHBlcmYgJT4lIHNlbGVjdCgtY29uZiksIGNhcHRpb24gPSBjYXAsIGRpZ2l0cyA9IDIsICJodG1sIikgJT4lCiAgICAgICAga2FibGVfc3R5bGluZyhmdWxsX3dpZHRoID0gRikpCn0KZm9yKHN0YXQgaW4gc3RhdHNfcGVyX2NvcmUpIHsKICAgIHN0YXRfdW5pdHMgPC0gdW5pdHMkdW5pdHNbdW5pdHMkbWV0cmljPT1zdGF0XQogICAgaWYoc3RhdF91bml0cz09IiIpIHsKICAgICAgICBjYXAgPC0gc3RhdAogICAgfSBlbHNlIHsKICAgICAgICBjYXAgPC0gcGFzdGUwKHN0YXQsICIsICIsIHN0YXRfdW5pdHMpCiAgICB9CiAgICBjYXQocGFzdGUwKCI8aDQ+IixjYXAsIjwvaDQ+XG4iKSkKICAgIHBlcmYgPC0gaHBjYyAlPiUgZmlsdGVyKG1ldHJpYyA9PSBzdGF0KSAlPiUKICAgICAgICBncm91cF9ieShyZXNvdXJjZSxleGVfdHlwZSxjcHVzKSAlPiUKICAgICAgICBzdW1tYXJpc2UoYXZlcmFnZT1tZWFuKHZhbHVlKmNwdXMpLHN0ZGV2PXNkKHZhbHVlKmNwdXMpKSAlPiUKICAgICAgICB1bmdyb3VwKCkgJT4lIG11dGF0ZShjb25mPXBhc3RlKHJlc291cmNlLGV4ZV90eXBlLGNwdXMsIHNlcCA9ICIsICIpKSAlPiUKICAgICAgICBtdXRhdGUoY29uZj1mYWN0b3IoY29uZiwgbGV2ZWxzID0gcmV2KGNvbmYpKSkKICAgIHByaW50KGdncGxvdChwZXJmLCBhZXMoeD1jb25mLCB5PWF2ZXJhZ2UpKSArCiAgICAgICAgZ2VvbV9iYXIoc3RhdD0iaWRlbnRpdHkiLCBmaWxsPSJyb3lhbGJsdWUiLCBjb2xvcj0icm95YWxibHVlNCIpICsKICAgICAgICBnZW9tX2Vycm9yYmFyKGFlcyh5bWluPWF2ZXJhZ2Utc3RkZXYsIHltYXg9YXZlcmFnZStzdGRldiksIHdpZHRoPS4yKSsKICAgICAgICBjb29yZF9mbGlwKCkreWxhYihjYXApKQogICAgcHJpbnQoa2FibGUocGVyZiAlPiUgc2VsZWN0KC1jb25mKSwgY2FwdGlvbiA9IGNhcCwgZGlnaXRzID0gMiwgImh0bWwiKSAlPiUKICAgICAgICBrYWJsZV9zdHlsaW5nKGZ1bGxfd2lkdGggPSBGKSkKfQpgYGAKCiMjIyBQZXJmb3JtYW5jZSBwZXIgQ29yZSAodGhhdCBpcyBNUEkgUHJvY2VzcykKCmBgYHtyIHJlc3VsdHM9J2FzaXMnLCBlY2hvPUZBTFNFfQpmb3Ioc3RhdCBpbiBzdGF0c19vdmVyYWxsKSB7CiAgICBzdGF0X3VuaXRzIDwtIHVuaXRzJHVuaXRzW3VuaXRzJG1ldHJpYz09c3RhdF0KICAgIGlmKHN0YXRfdW5pdHM9PSIiKSB7CiAgICAgICAgY2FwIDwtIHN0YXQKICAgIH0gZWxzZSB7CiAgICAgICAgY2FwIDwtIHBhc3RlMChzdGF0LCAiLCAiLCBzdGF0X3VuaXRzKQogICAgfQogICAgY2F0KHBhc3RlMCgiPGg0PiIsY2FwLCI8L2g0PlxuIikpCiAgICBwZXJmIDwtIGhwY2MgJT4lIGZpbHRlcihtZXRyaWMgPT0gc3RhdCkgJT4lCiAgICAgICAgZ3JvdXBfYnkocmVzb3VyY2UsZXhlX3R5cGUsY3B1cykgJT4lCiAgICAgICAgc3VtbWFyaXNlKGF2ZXJhZ2U9bWVhbih2YWx1ZS9jcHVzKSxzdGRldj1zZCh2YWx1ZSkpICU+JQogICAgICAgIHVuZ3JvdXAoKSAlPiUgbXV0YXRlKGNvbmY9cGFzdGUocmVzb3VyY2UsZXhlX3R5cGUsY3B1cywgc2VwID0gIiwgIikpICU+JQogICAgICAgIG11dGF0ZShjb25mPWZhY3Rvcihjb25mLCBsZXZlbHMgPSByZXYoY29uZikpKQogICAgcHJpbnQoZ2dwbG90KHBlcmYsIGFlcyh4PWNvbmYsIHk9YXZlcmFnZSkpICsKICAgICAgICBnZW9tX2JhcihzdGF0PSJpZGVudGl0eSIsIGZpbGw9InJveWFsYmx1ZSIsIGNvbG9yPSJyb3lhbGJsdWU0IikgKwogICAgICAgIGdlb21fZXJyb3JiYXIoYWVzKHltaW49YXZlcmFnZS1zdGRldiwgeW1heD1hdmVyYWdlK3N0ZGV2KSwgd2lkdGg9LjIpKwogICAgICAgIGNvb3JkX2ZsaXAoKSt5bGFiKGNhcCkpCiAgICBwcmludChrYWJsZShwZXJmICU+JSBzZWxlY3QoLWNvbmYpLCBjYXB0aW9uID0gY2FwLCBkaWdpdHMgPSAyLCAiaHRtbCIpICU+JQogICAgICAgIGthYmxlX3N0eWxpbmcoZnVsbF93aWR0aCA9IEYpKQp9CmZvcihzdGF0IGluIHN0YXRzX3Blcl9jb3JlKSB7CiAgICBzdGF0X3VuaXRzIDwtIHVuaXRzJHVuaXRzW3VuaXRzJG1ldHJpYz09c3RhdF0KICAgIGlmKHN0YXRfdW5pdHM9PSIiKSB7CiAgICAgICAgY2FwIDwtIHN0YXQKICAgIH0gZWxzZSB7CiAgICAgICAgY2FwIDwtIHBhc3RlMChzdGF0LCAiLCAiLCBzdGF0X3VuaXRzKQogICAgfQogICAgY2F0KHBhc3RlMCgiPGg0PiIsY2FwLCI8L2g0PlxuIikpCiAgICBwZXJmIDwtIGhwY2MgJT4lIGZpbHRlcihtZXRyaWMgPT0gc3RhdCkgJT4lCiAgICAgICAgZ3JvdXBfYnkocmVzb3VyY2UsZXhlX3R5cGUsY3B1cykgJT4lCiAgICAgICAgc3VtbWFyaXNlKGF2ZXJhZ2U9bWVhbih2YWx1ZSksc3RkZXY9c2QodmFsdWUqY3B1cykpICU+JQogICAgICAgIHVuZ3JvdXAoKSAlPiUgbXV0YXRlKGNvbmY9cGFzdGUocmVzb3VyY2UsZXhlX3R5cGUsY3B1cywgc2VwID0gIiwgIikpICU+JQogICAgICAgIG11dGF0ZShjb25mPWZhY3Rvcihjb25mLCBsZXZlbHMgPSByZXYoY29uZikpKQogICAgcHJpbnQoZ2dwbG90KHBlcmYsIGFlcyh4PWNvbmYsIHk9YXZlcmFnZSkpICsKICAgICAgICBnZW9tX2JhcihzdGF0PSJpZGVudGl0eSIsIGZpbGw9InJveWFsYmx1ZSIsIGNvbG9yPSJyb3lhbGJsdWU0IikgKwogICAgICAgIGdlb21fZXJyb3JiYXIoYWVzKHltaW49YXZlcmFnZS1zdGRldiwgeW1heD1hdmVyYWdlK3N0ZGV2KSwgd2lkdGg9LjIpKwogICAgICAgIGNvb3JkX2ZsaXAoKSt5bGFiKGNhcCkpCiAgICBwcmludChrYWJsZShwZXJmICU+JSBzZWxlY3QoLWNvbmYpLCBjYXB0aW9uID0gY2FwLCBkaWdpdHMgPSAyLCAiaHRtbCIpICU+JQogICAgICAgIGthYmxlX3N0eWxpbmcoZnVsbF93aWR0aCA9IEYpKQp9CmBgYAo=